Can Observational Data Replace Randomized Trials?

TO THE EDITOR: The increasing complexity and cost of conducting randomized trials have stimulated interest in using observational data sets to evaluate cancer treatments. Determining the causal effect of treatments from observational data is, however, challenging because more aggressive treatments are selectively prescribed for patients with adverse disease characteristics or favorable comorbidity profiles. Associations may, therefore, arise between treatments and outcomes that are the result of confounding and are not causal. Relatively little is known about the extent of such confounding or the degree to which it can be removed through stratification by prognostic variables. Breast cancer is one of the commonest conditions for which radiotherapy is prescribed. We have therefore used it to examine this issue. We analyzed data on women registered between 1990 and 2008 in the SEER public-use data set. Women were excluded if they were younger than 20 or older than 80 years when diagnosed, had previous cancer, unknown cancer laterality, bilateral cancer, or unknown radiotherapy status. Each woman entered the study on the date of her breast cancer diagnosis and left on the earliest of the following events: death, loss to follow-up, turning age 85 years, or January 1, 2009. Two analyses were conducted. In the first, deaths and person-years were stratified by five basic variables; the second stratification also included all available prognostic variables. Mortality ratios were estimated by maximum likelihood using Poisson regression. Calculations were performed using STATAversion 12 (STATA, College Station, TX). Information was also collated from the Early Breast Cancer Trialists’ Collaborative Group (EBCTCG) meta-analyses of randomized trials of radiotherapy versus not. Analyses were stratified by trial, individual follow-up year, age at randomization, and nodal status. After breast-conserving surgery, with only basic variables in the stratification, radiotherapy was associated with reduced mortality for all the causes examined (Appendix Table A1, online only). Stratifying for all available prognostic variables changed the estimates, but all remained significantly below one. Notably, the rate ratio for mortality from all causes except breast cancer (0.69) was lower than that for breast cancer (0.74). After mastectomy in node-positive disease, with only basic variables in the stratification, radiotherapy was associated with increased breast cancer mortality and all-cause mortality, and decreased mortality from all causes except breast cancer. Stratifying for prognostic variables changed the death rate ratios for breast cancer from 1.35 to 0.89 and all causes of death from 1.21 to 0.85, but the rate ratio for mortality from all causes except breast cancer (0.74) was still lower than that for breast cancer (0.89). Estimates of the effects of radiotherapy in the EBCTCG meta-analyses of randomized trials were then compared with the corresponding death rate ratios in the SEER data stratified by all available variables (basic and prognostic; Fig 1). After breastconserving surgery, the breast cancer death rate ratio in the EBCTCG data was 0.83 for all women and did not differ significantly between pN0 (node-negative disease) and pN1 (node-positive disease; P5 .76), whereas in the SEER data the corresponding death rate ratio for all women was lower (0.74 v 0.83; P 5 .04), and differed significantly between pN0 and pN1 (0.84 v 0.73; P 5 .008). After mastectomy, the breast cancer death rate ratio in the EBCTCG data was 0.85, similar to that after breast-conserving surgery, and it did not differ significantly according to the number of positive nodes (P for heterogeneity 5 .53). In contrast, in the SEER data, there was substantial heterogeneity in the breast cancer death rate ratios according to the number of positive nodes (P for heterogeneity , .001). In women with one to three positive nodes, postmastectomy radiotherapy was associated with significantly increased breast cancer mortality in the SEER data (1.10; 95%CI, 1.02 to 1.18) and significantly decreased breast cancer mortality in the EBCTCG data (0.80; 95% CI, 0.67 to 0.95). For mortality from all causes except breast cancer, there were also major qualitative differences between the EBCTCG and SEER data. In the EBCTCGdata, radiotherapywas associatedwith significantly higher rates of mortality from all causes except breast cancer, including from heart disease and from lung cancer. In the SEER data, radiotherapy was associated with significantly lower rates of mortality from all causes except breast cancer, including from heart disease. SEER is one of the largest, most detailed data sets. If we had used these SEER analyses to draw conclusions about the causal effects of radiotherapy, we would have concluded that radiotherapy after breastconserving surgery is more effective in node-positive than in nodenegative disease, and that radiotherapy after mastectomy in women with one to three positive nodes causes death from breast cancer. We would have also concluded that radiotherapy prevents mortality from all causes except breast cancer, including from heart disease and from accidents and violence (Appendix Table A1). These results contradict those of the randomized trials. We conclude, as have others, that nonrandomized comparisons are liable to provide misleading estimates of treatment effects. Therefore, they need careful justification every time they are used.